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Using a Video Split-Screen Technique to Evaluate Streaming Instructional 

Videos 

William J. Gibbs 
Ronan S. Bemas 
Steven A. McCann 
Eastern Illinois University 

Abstract 

The Media Center at Eastern Illinois University developed and streamed on the Internet 26 short (1-5 minutes) instructional 
videos about WebCT that illustrated specific functions, including logging- in. changing a password, and using chat. This study 
observed trainees using and reacting to selections of these videos. It set out to assess attitudes toward the quality of online videos 
and to identify perceptions trainees had about the video impact on their learning. A secondary aim of the study was to evaluate 
the potential of a video-split-screen technique for making observations of trainees during training. 

Trainees responded positively to the video training and the mode of delivery. They also perceived their learning to have been 
positively impacted as a result of it. As an observation tool, the video-split-screen technique was useful and it yielded much data 
in multiple media formats. 

Instructional video training as presented in this paper appears to be a viable WebCT resource. An extensive development 
effort was not needed to produce a worthwhile product that is easily modified and updated. These facts coupled with trainees’ 
positive reactions suggest that this instructional modality can be an effective supplement to face-to-face training. 

Introduction 

In 1997, the Media Center at Eastern Illinois University embarked on a technology training initiative aimed at improving 
faculty members’ ability to effectively and appropriately utilize and integrate computer technology into the teaching and learning 
process. To foster faculty members’ willingness to invest in using technology , the initiative provided services in three key areas. 
First, technical support services addressed software and hardware problems. For instance, an instructor having difficulty with 
HTML coding or a specific software application could call the Center for assistance. Second, the Center developed courseware 
applications as well as materials to support classes, including Web sites and CD-ROM materials. Third, the Center offered a 
number of faculty development activities, such as hands-on training workshops, informational sessions/demonstrations, and 
computer-based tutorials. In this paper, the authors limit the discussion of the initiative to the area of faculty (technology) 
development. 

Subsequent to the implementation of the initiative the University adopted WebCT as its Web-based course development 
platform. WebCT is a development tool that enables instructors to create and distribute on-line class materials, or entire online 
courses. The University administration charged the Media Center with formulating a WebCT training program, as part of it 
existing training initiative. The program was to introduce the software to faculty and teach them how to use it for instructional 
and learning purposes. 

The Center’s staff delivered WebCT training primarily through face-to-face workshops that were 23 hours in length. 
Throughout each academic semester, they offered over 40 workshops on the following topics: 

• WebCT basics: An Introduction to WebCT 

• Using WebCT Communication Tools 

• Using WebCT Quiz Modules 

• Using WebCT Student Management 

• Using WebCT File Manager, Course Content, Calendar 

Several open sessions were also offered in which the staff presented no formal instruction but assisted faculty with developing 
content in WebCT. 

Streaming Training Content 

While the WebCT trainers successfully offered numerous workshops at varying times throughout the semester to 
accommodate faculty schedules, they observed three shortcomings with the workshop approach. First, despite the availability of 
sessions, class scheduling conflicts prevented some faculty from attending. Second, faculty indicated that the timing of the 
workshops did not coincide with the time they allocated for development. For example, one individual commented that while he 
attended a workshop in October, he was unable to do any WebCT development until the end of the semester, at which time, he 
thought his familiarity with the program would wane. Third, in many ways, the workshops did not support individual work habits 
or needs. Trainers offered all sessions during normal work hours and training was general in nature so that it accommodated the 
greatest number of participants. As a result, an individual working outside normal business hours could not call on the expertise 
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of a trainer and was therefore left to his/her own resources to solve a problem. Moreover, individuals with specific WebCT needs 
or questions were difficult to accommodate during workshops, and the trainers observed that many of these unique needs were 
unmet. 

The aforementioned issues associated with the workshop format prompted the Media Center staff to explore alternative 
training delivery modalities, one of which being streaming video training. Two staff instructional technologists developed 26 
short (1-5 minutes) instructional videos about WebCT that illustrated specific functions, including logging-in, changing a 
password, and using chat. While the videos were general to WebCT, they contained information specific to Eastern Illinois 
University, such as how to access a WebCT course from the university home page or how to obtain a WebCT account. The 
technologists used a screen capturing utility (Snagit) to record the WebCT screens, and they narrated while performing program 
functions (see Figure 1). 
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Figure L WebCT Video Played in a Web Browser 

The approach did not require many hours of development, an important criterion of the technologists. Time constraints prohibited 
these individuals from spending exceedingly long periods of time developing the videos. Moreover, the changing nature of the 
WebCT software with updated versions, etc. necessitated a short development cycle. 

Developing the WebCT Training Videos 

As mentioned, the instructional technologists used the Snagit software utility to capture computer video screens and audio 
narration simultaneously. The program records the video and audio into a single Avi file, which is commonly used in Windows- 
based PC’s. 
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The recordings produced large file sizes that could not be streamed on the Internet and so the Avi files were converted to a 
QuickTime format. QuickTime was chosen because the Media Center owned a QuickTime streaming server. The accepted 
industry standard for converting audio and video files with the least possible quality loss (or generation loss) is Terran 
Corporation’s Media Cleaner Pro. Version 5 of the software is called Cleaner. Cleaner and QuickTime Pro both encode 
miscellaneous QuickTime and streaming QuickTime files with constant bit -rate encoding using several different types of 
compression schemes. 

Compressing video causes the video to shrink in size. Instead of encoding each frame of video with the entire contents of that 
frame, only the changes from frame to frame are encoded. In addition, if colors in an area of the video are similar, a single byte of 
information representing the entire area is used resulting in smaller file size. Smaller video files and lower data rates enable the 
video to be streamed more easily over the Internet and to be viewed on dated computers. 

The Media Center also purchased a plug-in for Media Cleaner Pro called Sorenson Developer edition. In order to encode a 
QuickTime movie with variable bit-rate encoding the plug-in is required. Variable bit -rate encoding allows the bit-rate to vary as 
the file requires bytes resulting in higher quality video at a low average bit -rate. Constant bit -rate provides predictable data rates, 
but produces video of poorer quality. 

Typically, the largest video frame size that can be streamed over the Internet or a very fast connection (e.g., T-1 or faster) is 
320 X 240 with 10 frames per-second. Since the size of the WebCT videos was approximately 640 X 400, a compromise was 
achieved by decreasing the frame rate. To obtain the same data rate, one can increase frame size while decreasing frame rate or 
vice versa. Since the motion on the screen was mainly mouse movement and Web page scrolling, the file frame rate was 
decreased to 4 frames per-second. 

After the files were processed by Cleaner 5, they were copied to the QuickTime server. The developers used a program called 
Make Reference Movie to create small QuickTime files (approximately 4K) that were placed on the university’s web server 
along with HTML documents that linked to the videos. 

Purpose 

In the researchers’ view, delivering the videos in a Web-based hypermedia environment was unique, and it presented a 
number of issues that needed careful examination. As a result, they set out to examine how trainees would use and react to the 
instructional videos. Specifically, the researchers set out to: 

1 . assess trainees’ attitudes about the quality of the online instructional videos used in the project and the method by 
which they were delivered; 

2. identify trainees’ perceptions about the effectiveness of the videos to fostering their learning. 

A secondary aim of the study was to evaluate the potential of a video split-screen technique for observing the processes (e.g., 
performance tasks/work behaviors) trainees engage in while using the instructional videos. 



Method 

Hypermedia, characterized by an arrangement of nodes and links, provides non-sequential access to mediated content 
(Kumar, Helgeson & White, 1994). These informationally rich and flexible user- centered designs add complexity to the study of 
how users interact with a system (Gay & Mazur, 1993). Some evaluation approaches promote a holistic orientation (Winograd & 
Flores, 1986) using qualitative methods (Card, Moran & Newell, 1983) and multiple data collection instruments (Marchionini, 
1990). Instruments for monitoring human-computer interactions often allow for the compilation of data in visual (video, 
photographic), textual, and auditory form. 

In accordance with a more qualitative approach to evaluation, the researchers used a video-split-screen recording technique to 
monitor how trainees used the instructional videos. A video camera recorded trainees as they used the software. Simultaneously, 
a scan converter converted the computer screen output to a video/NTSC signal. Both signals (trainee and computer screen) were 
sent to a video effects generator and combined into one image, which allowed the researchers to observe the computer screen, 
including mouse movements, object (e.g., buttons, links, etc.) selections, and web page changes simultaneously with trainees’ 
behavior (see Figure 2). 
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Figure 2. Video Split-screen Recording Process 



Materials 

The researchers created a web site that provided an overview of the study and instructions for completing it. They chose 1 1 
of the 26 instructional videos for review because the selected videos represented fundamental WebCT components. Video topics 
included: 

• Logging into WebCT 

• Modifying headers and footers 

• Communication Tools Overview Part 1 

• Communication Tools Overview Part 2 

• WebCT Mail Overview 

• Sending Mail through WebCT 

• Using Bulletin Boards Part 1 

• Using Bulletin Boards Part 2 

• Using Bulletin Boards Part 3 

• Starting Chat 

• Using Chat 

Prior to beginning the study, trainees completed a pre-usability survey that assessed their familiarity with various WebCT 
components. Trainees rated their familiarity on a 7-point scale from 1 (Completely Unfamiliar) to 7 (Very Familiar). They 
viewed each of the 1 1 videos, in any order of preference. After viewing each video, trainees practiced the WebCT function or 
component they just viewed. When practicing, the researchers asked them to think aloud or verbalize what was going through 
their minds. After practicing, trainees completed a short WebCT Function survey, which presented 7 attitudinal statements to be 
rated from 1 (Strongly Disagree) to 7 (Strongly Agree). Once trainees watched all 1 1 videos, practiced them, and completed their 
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respective function surveys, they received a post-usability survey that presented identical questions to those of the pre-usability 
survey. It took each trainee approximately two hours to complete the study. 

The five trainees who completed all evaluation tasks were undergraduate students employed at the Media Center. Most 
indicated that they were computer familiar but had little or no experience with WebCT. 

Results 

Table 1 shows how familiar the trainees were of the WebCT components before and after the video training. The higher the 
average rating, the more familiar the trainees were with the specific component (the scale ranges from I to 7). Separate t-tests for 
dependent or paired means were conducted for each WebCT component. At a significance level of .05, results indicate that the 
trainees became significantly more familiar with the components after the video training. Table I also shows that the trainees 
were slightly familiar with the WebCT in general and with logging into the WebCT before the training, but were completely 
unfamiliar with the specific WebCT components. 



WebCT Component 


Before Video Training 


After Video Training 


WebCT in general 
Logging into WebCT 
Changing WebCT password 
Modifying header/footer on the WebCT 
homepage 

Reading a message in a WebCT discussion 
forum 


2.60 

3.60 

1.80 

1.40 


5.60 

6.80 

5.20 

7.00 


1.40 


5.80 


Posting a message to a WebCT discussion 
forum 


1.40 


5.80 


Reading a message with WebCT email 


1.40 


6.40 


Posting a message with WebCT email 


1.40 


6.40 


Accessing WebCT chat rooms 


1.40 


6.40 


Using WebCT chat rooms 


1.40 


6.40 



Table L Familiarity with WebCT s Components Before and After the Video Training 

The trainees' evaluations of the eleven videos were averaged per evaluation item and are shown in Table 2 below. For each item, 
the trainees gave very positive evaluations of the videos. The higher the average rating, the more positive the evaluation (on a 
scale of 1 to 7). 



Evaluation Item 



Average Rating 



No problems accessing the video 6.76 

No problems using and viewing the video 5.95 

The video was easy to understand 6.33 

The video was well-paced 6.44 

The video helped in learning about the specific WebCT function 6.59 

The video format was the best way of learning about the specific WebCT function 6.62 

Would recommend that others use the specific video 6.40 



Table 2. Trainees ' Evaluations Across the Training Videos 

The trainees' evaluations of each instructional video were obtained by averaging their ratings across the seven evaluation 
items. The average ratings per video are shown in Table 3 below. The trainees gave very positive evaluations for each video. 
The higher the average rating, the more positive the evaluation (on a scale of 1 to 7). 
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Training Video Title 


Average Rating 


Logging into WebCT 


6.11 


Modifying headers and footers 


6.17 


Communication tools overview (part 1) 


6.29 


Communication tools overview (part 2) 


6.54 


WebCT mail overview 


6.69 


Sending mail through WebCT 


6.69 


Using bulletin boards (part 1) 


6.37 


Using bulletin boards (part 2) 


6.57 


Using bulletin boards (part 3) 


6.40 


Starting chat 


6.51 


Using chat 


6.49 



Table i. Trainees ' Evaluations of the Specific Training Videos 

Trainees Commentary 

Overall, trainees had few general comments about the videos. Two individuals wrote the following comments on the pre- 
usability survey: 

• I did a little with WebCT questions over in the Media Center. 

• I don't know anything about WebCT so it will be nice to learn. 

The same two individuals wrote the following comments on the post-usability survey: 

• I don't feel I learned much. 

• This was fairly easy to understand. Because the video was sometimes fuzzy it was hard to see everything that was going 
on. I feel like I have a good grasp of the beginnings of WebCT and could probably figure out more things on my own. 

It is difficult to ascertain whether the trainee who wrote, I did not learn much, was commenting about the inability of the video 
training to facilitate learning about WebCT or that her learning did not increase because her skills equaled the level of mastery 
the videos sought to promote. Her ratings of post -usability survey items were high (M = 6.4) compared to her pre-usability survey 
ratings (M = 1.7), which suggests that she perceived her learning to have increased. 

Trainees made 15 independent comments about the videos (see Table 4). Generally, the commentary reflected their concerns 
about the video images appearing fuzzy. Internet congestion periodically caused the video image to distort. Comments also 
point out that trainees preferred the videos to be short, helping them to avoid feeling overwhelmed. Lastly, commentary reflected 
that some trainees saw similarities between specific WebCT functions and third-party email software and, for at least one 
participant, this association fostered understanding of WebCT functions. 
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Training Video Title 


Comments 


Logging into WebCT 


Improve the picture quality of the video. It was choppy and messy as it ran. 
It was difficult at time to see what was actually on the screen. It was fuzzy a 
few times but because the voice told me what was there, I knew what to 
expect. If possible it may be quicker to one-on-one demonstrations. 


Modifying headers and footers 


Good timing of the videos. They are not too long and don't make me feel 
overwhelmed. 

Because the screen is fuzzy it was hard to catch all of the information. 
There were several screens that were similar, but it didn't take too long to 
figure out. 


Communication Tools Overview 
Part 1 


This was less information to soak in. This was easier than the previous one. 


Communication Tools Overview 
Part 2 


This was easy to understand because it was just telling about the links and 
how they are similar to other things we use everyday. 


Web CT Mail Overview 


I think this is very basic, and I would expect most people to know it, but if 
they didn't this would help them. 

Again, this was easy to understand because it was an overview and because 
it works basically the same way email does. 


Sending Mail through Web CT 


This was easy to understand because it works Just like regular email. 
Anyone who has used email will be able to understand it. 


Using Bulletin Boards Part 1 


Good timing on the videos. They are not long and don't make me feel 
overwhelmed. 

Because this is similar to email it is very easy to understand. It's still fuzzy 
at times. 


Using Bulletin Boards Part 2 


This was short and easy to understand. 


Using Bulletin Boards Part 3 


This worked well because it was explained twice. I could see what I was 
doing and it was immediately reflected on the screen. 


Starting Chat 


I have never been in or used a chat room so I don't know if this is similar but 
it was easy for me to understand. 


Using Chat 


This was easy to understand as well. It was similar to email and other 
functions on WebCT. 



Table 4. Trainee Commentary About Instructional Videos 

Discussion 

Trainees’ Use of the Videos 

The researchers were in interested in examining how trainees used the instructional videos. They speculated that trainees 
might watch them while working in WebCT. For example, a trainee developing content in WebCT could open two browser 
windows, one of which contained a video and the other the WebCT workspace. He could then watch a video as needed while 
working. 

Overall, trainees did not use the videos when working. They watched a video and, after it ended, opened the WebCT 
workspace to perform tasks without returning or reviewing the video. In a few isolated instances, they left the WebCT workspace 
to review a video only when an alternative solution to a task could not be found. When encountering a problem, trainees 
attempted to find a solution within the WebCT workspace (e.g., clicking WebCT options, etc.) prior to reviewing a video. 

Trainees’ Reactions to the Videos 

Trainees perceived that their learning increased as a result of the instructional videos. From the video recordings, the 
researchers made two observations that appear to support this perception. First, trainees, when performing WebCT tasks, did so 
in the same sequence and using the same input information as presented in videos. Second, there are several functions in WebCT 
that, in the researchers’ view, are not obvious to users. When the videos presented such tasks, trainees performed them. For 
example, the chat room video demonstrated how to send a private chat message, a feature not always apparent to users. When 
trainees performed tasks associated with the chat room, they each sent a private message. Had trainees not viewed the video, 
they would not have known about this task and how to perform it. 

Trainees’ perception that they learned from the videos, their high ratings of them, and the researchers’ observation pertaining 
to the potential of the videos to support learning suggests that this mode of WebCT training is beneficial. Moreover, the videos 
can be created in a short development cycle without excessive time demands being placed on the developers. 
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The Video Sjlit-Screen Evaluation Method 

The researchers observed that the video-spit-screen evaluation technique provided several advantages. It enabled them to 
trace (visually and aurally) the actions taken by trainees and to monitor their use of the Web-based videos, including time on task, 
navigation of landscape, and options selected. At one point, a trainee could not get a WebCT chat room to open. Using the video, 
the researchers retraced the trainee’s actions. There were able to view his verbal and non-verbal response and frustration with 
being unable to open the room. They reconstructed the actions he took attempting to open the room and the length of time he 
spent on this task before asking for help. They were also able to observe distractions or the series of actions that led to 
unsuccessful results and to make informed judgments (based on verbalizations) about the trainee’s decision making. 

The researchers felt that the video-split -screen technique was a useful data collection method but they noted three 
disadvantages. First, it requires evaluators to invest much time in data analysis. The split-screen method collects a large amount 
of data in multiple media formats. Proper analysis of them requires time, which may be prohibitive for some Web developers. 
Second, the approach requires specialized video and computing hardware that some developers do not own. Third, untrained 
evaluators need guidance or training in how to conduct evaluations. Only after a review of the video recordings did the 
researchers realized the extent to which their comments served as prompts to trainees and, in some cases, directed their actions. 
Untrained evaluators must ensure their questioning and prompting do not lead users. 

Summary and Further Research 

As described in this paper, the instructional video training appears to be a viable WebCT resource. An extensive 
development effort 

was not needed to produce a worthwhile product that is easily modified and updated. These facts coupled with trainees’ positive 
reactions to them suggest that the videos hold potential for training. 

All the videos used in this study were streamed, which worked well since trainees had fast Internet connections. However, as 
training audiences extend to off-campus locations, slow Internet connections will, to a great extent, prohibit video training. 
Alternative delivery formats such as DVD and CD-ROM will be needed to reach geographically dispersed trainees. 

This study was an initial attempt to observe trainees using and reacting to selected instructional videos and, in this regard, the 
video-split-screen technique yielded much data. The work presented here needs to be extended to more fully analyze the 
information collected and its implications for the design and delivery of video training. The video-split-screen technique presents 
the possibility of making more in-depth analysis of not only software evaluations but trainees’ information processing and 
decision making as they are engaged with software. 
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